Skip to content

[trainer correctness] Backport Miles' fix for gradients kept between forward_backward calls - #257

Merged
Micah Yong (micahtyong) merged 9 commits into
mainfrom
micah/nemotron-perf/trainer-expert-grad-accumulation
Nov 17, 2026
Merged

Micah Yong (micahtyong) merged 9 commits into
mainfrom
micah/nemotron-perf/trainer-expert-grad-accumulation

Conversation

@micahtyong

@micahtyong Micah Yong (micahtyong) commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

A Tinker client can call forward_backward several times before one optim_step, and several clients' adapters can share one trainer. Miles keeps each adapter's gradients until its optim_step. Megatron ends every backward pass by combining gradients across GPUs over the whole gradient buffer, including the gradients kept from earlier calls. For most weights, combining them again changes nothing. For two kinds it does:

  • Routed-expert adapters. Their buffer is divided by the data-parallel size but summed over the GPUs that hold copies of the same experts, so kept gradients come back multiplied by tensor_model_parallel_size / (expert_model_parallel_size × expert_tensor_parallel_size).
  • Adapters on the MLA down-projections (q_a_proj, kv_a_proj_with_mqa). Their gradients are summed over the tensor-parallel GPUs, so kept gradients come back multiplied by tensor_model_parallel_size.

Each later forward_backward on a trainer therefore rescales the gradients waiting for an optim_step, whether it comes from the same client or another one. One forward_backward per optim_step with no other client in between was always correct. Recipes affected: DeepSeek-V3.1, Kimi-K2.6, GLM-5.3 and GLM-5.3 Flash, gpt-oss-120b 128k, gpt-oss-20b 64k, Nemotron-3.5-Lightning 256k, Nemotron 3 Super and Ultra, Qwen3.5-397B and Qwen3.6-35B-A3B.

Upstream Miles fixed this in radixark/miles#4480, and the miles-v0.2.0 image already has that fix. This PR backports the same function, unchanged, for the default Miles images: before each training pass it sets the kept gradients aside, so the pass's reduce only sees the new gradients, then adds them back. It is installed only for the default images and goes away when they move to Miles v0.2.0. It holds one extra copy of the gradient buffers during each training call, so it only runs on layouts where the reduce would rescale kept gradients: experts sharded differently from tensor parallelism, or sequence-parallel adapters with tensor parallelism above 1. Other layouts, and forward-only calls, are unchanged.

This replaces #221, which fixes the same problem the same way.

E2E results

Setup: Nemotron-3.5-Lightning 30B weights on H200s, layout set by recipe overrides, LoRA rank 32, stock Tinker SDK. One client makes k identical forward_backward calls and then one optim_step. The table shows the accumulated gradient divided by the single-call gradient, which should be k.

Layout Before WandB After WandB Routed experts, k = 2 Routed experts, k = 4
TP2 / EP1, 2 GPUs Before After 3.00 → 2.01 14.93 → 4.01
TP2 / EP4 / DP2, 4 GPUs Before After 1.50 → 2.00 1.87 → 4.00
TP2 / EP2, 2 GPUs (stock recipe) – After – 3.98 → 4.01

Attention and shared-expert adapters were already right and stay at 2.00 and 4.00. With two clients, client A's routed-expert update after one call from client B was 1.99 times what it should be on TP2 / EP1; with this PR it is 1.00, also after two calls from B, after B's own optim_step, and after forward-only calls.

The extra copy costs memory and time. On TP2 / EP1 with four adapter slots it is 12.85 GB per GPU, and 128 rows of 1,600 tokens take 10.79 s in forward_backward instead of 9.06 s (before, after). The stock TP2 / EP2 layout is not affected and skips the copy.

Megatron ends every backward pass by reducing the gradient buffers. Miles
keeps gradients until the optimizer step, so gradients from earlier
forward_backward calls go through those reduces again. Two of them change
the kept gradients:

- Each buffer is multiplied by 1 / data-parallel size and summed over its
  data-parallel group. Dense buffers come back unchanged. Expert buffers
  are summed over the expert data-parallel group, so they come back
  multiplied by tensor_model_parallel_size / expert_model_parallel_size.
- Weights replicated across tensor-parallel ranks, such as adapters on
  the MLA down-projections, are summed over those ranks, so they come
  back multiplied by tensor_model_parallel_size.

Any forward_backward on the trainer triggers it, including another
client's, because the reduces cover every adapter slot.

Divide the kept gradients by those factors before each backward pass, so
the reduces return them unchanged. A failed forward_backward marks the
trainer unusable, so there is no partly scaled state to restore.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
keep_grads_through_reduce has to match every reduce that ends a backward
pass. A reduce it misses would rescale kept gradients without any error.
Each rank now reduces gradients of all ones once when the trainer starts,
through the same call a backward pass makes, and fails if any gradient
comes back changed. The error names the first weight that changed.

The gradient accumulation tests now check one invariant per layout: kept
gradients come back from the reduce unchanged.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Merge main and replace the pre-divide and startup check with
radixark/miles#4480's accumulate_gradients, which the miles-v0.2.0 image
already runs: set kept gradients aside for each training pass so its reduce
only sees the new gradients, then add them back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@micahtyong Micah Yong (micahtyong) changed the title [trainer correctness] Keep gradients unchanged between forward_backward calls [trainer correctness] Backport Miles' fix for gradients kept between forward_backward calls Nov 16, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rainer-expert-grad-accumulation

# Conflicts:
#	CHANGELOG.md
# Conflicts:
#	src/spindle/backends/miles_runtime/actor.py
# Conflicts:
#	src/spindle/backends/miles_runtime/actor.py
@micahtyong
Micah Yong (micahtyong) merged commit d7f8c91 into main Nov 17, 2026
3 checks passed
@micahtyong
Micah Yong (micahtyong) deleted the micah/nemotron-perf/trainer-expert-grad-accumulation branch November 17, 2026 02:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant